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ABSTRACT 

A  single  server  queueing  system  with  Poisson  arrival 
at  rate  X  and  a  finite  queue  capacity  N  is  considered. 

The  service  can  be  performed  using  one  of  K  available 
service  rates  y^  ^  —  •  ♦  •  —  ^  *  When  service  rate  y^ 

is  in  effect  the  service  time  is  a  random  variable  exponentially 
distributed  with  expected  service  time  1/y^  »  and  a  cost  rate 
c(y^)  is  incurred.  For  each  customer  served,  a  reward  of 
A  will  be  earned.  The  problem  is  to  choose  service  rate  in 
order  to  minimize  the  expected  discounted  or  expected  long 
run  average  cost.  We  first  formulate  the  problem  as  a  semi- 
Markov  decision  process  with  the  state  as  the  number  of 
customers  in  the  system.  It  is  then  shown  that  the  optimal 
policy  is  switchover  or  reverse  switchover  depending  upon 
whether  or  not  the  system  is  profitable.  A  switchover  policy 
is  a  policy  which  uses  higher  rates  in  the  higher  states  and  a 
reverse  switchover  policy  uses  lower  rates  in  the  higher  states. 
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INTRODUCTION 


1.1  Description  of  Problem 

Optimization  problems  in  queueing  have  received  a  large  amount  of 
attention  during  the  past  few  years.  Up  to  a  decade  ago  most  of  the  works 
in  queueing  were  concerned  about  the  descriptive  aspect  of  the  problem , 
namely  deriving  mathematical  characteristics  and  structure  of  the  system 
such  as  stationary  dist ributions ,  waiting  time  distribution,  expected  number 
of  people  in  the  system,  etc.  In  almost  all  branches  of  industries  where 
queueing  plays  a  role,  in  addition  to  the  descriptive  models,  the  manager  is 
confronted  with  the  optimization  of  design  and  control  of  the  system.  With 
the  introduction  of  Markovian  decision  processes  and  its  relation  with 
queueing  the  optimization  of  queueing  system  gained  a  new  momentun. 

In  a  general  optimization  problem,  choosing  a  policy  or  value  of  a 
variable  in  order  to  minimize  a  function  is  sought.  In  queueing  these 
variables  can  enter  in  the  arrival  stream  or  service  facility.  The  latter 
is  encountered  more  in  practice  and  therefore,  more  attention  is  given  to 
these  problems,  but  a  number  of  works  can  be  found  which  controlling  the 
arrival  process  is  the  target  of  the  optimization.  In  some  case  even 
controlling  both  arrival  and  service  variables  have  been  the  target. 

In  this  thesis  we  are  concerned  with  the  second  type  of  problem.  Namely, 
the  decisions  are  made  on  the  service  facility. 

Ue  are  considering  a  queueing  system  with  a  single  servicing  facility 
and  a  limited  space  N  for  waiting  customers.  Those  customers  who,  upon 
arrival,  find  the  system  with  N  persons  do  not  stop.  The  arrival  stream 
Is  a  Poisson  process  with  rate  X  .  The  service  can  be  performed  with  one 
of  the  K  existing  service  rates  <_  u2  1  •••  —  •  when  »ervice  with 

rate  is  used  the  service  time  is  a  random  variable  which  is  exponentially 
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distributed  with  expected  servir*.  .lme  1 , .  The  ;erver  can  switch  a 

service  rate  to  another  at  the  time  of  an  arrival  or  at  the  time  of  a  departure. 
These  switches,  if  made,  must  optimize  the  overall  expected  discounted  or 
average  cost  for  the  given  cost  structure.  The  cost  structure  considered  is 
as  follows, 

(c)  A  service  cost  rate  c(^)  «  the  cost  of  servicing  at  rate  . 

It  Is  assumed  that  c(y^)  is  nondecreasing  in 

c(y.)  <,  c(p2)  ...  <  c(yR)  . 

(b)  The  server  will  receive  a  reward  A  for  each  customer  he  serves. 

With  this  cost  structure  the  optimal  policy  is  found  to  have  a  simple  form. 

If  the  state  of  the  system  is  defined  as  the  number  of  people  in  the  system, 
then,  e.g.,  in  the  case  of  two  rates  <  y^  »  will  be  shown  that  the 
optimal  policy  prescribes  up  to  a  state  J  ,  0  <  J  <  N  ,  and  u2  for  all 

states  higher  than  J  ,  which  we  will  call  a  switchover  policy.  The  case  of 
K  2  service  rates  is  also  analyzed  and  a  simple  policy  is  found. 

With  the  same  cost  structure  the  case  of  "hesitating  customers"  is  also 
analyzed,  where  the  arriving  customer  will  stop  with  a  probability  p^  if 
there  are  1  person  In  the  system.  It  is  shown  that  for  p.  monotone  the 

t  1 

same  type  of  policy  Is  optimal. 

Although  complete  results  are  not  found  for  the  expected  discounted 
model,  Chapter  II  la  devoted  to  this  model  since  some  of  the  results  in  this 
cate  are  needed  in  the  average  coat  case. 

1.2  Review  of  Literature 

Aa  was  mentioned  in  the  previous  section  the  problem  of  optimization 
In  queueing  has  been  given  a  considerable  amount  of  attention  during  recent 


years. 
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In  the  earliest  papers  assumptions  were  made  about  the  class  of  feasible 
policies  and  within  this  restricted  class  the  best  policies  were  found.  For 
example  in  Yadln  and  Naor  [18],  they  find  the  optimal  queue  size  at  which  to 
turn  the  server  on  assuming  that  the  policy  is  to  turn  the  server  on  when 
queue  length  reaches  a  certain  level. 

Heyman  [  3  ]  in  his  work  considered  the  model  of  Yadin  and  Naor  and 
attempted  to  find  the  form  of  an  optimal  policy  in  an  M/G/l  queue  without 
making  a  priori  assumptions  about  the  policy. 

Thatcher  [17]  consideres  an  M/G/l  queue  with  two  service  rates.  He  does 
not  assume  a  queue  limit  but  does  assume  a  holding  cost  which  is  dependent 
upon  the  number  of  customers  in  the  system.  He  first  finds  the  best  switch- 

t 

over  policy  and  then  he  proves  its  optimality  among  all  policies. 

In  general  when  the  form  of  optimal  policy  is  the  target,  the  theory  of 
stochastic  decision  processes  plays  an  important  role. 

As  was  mentioned  before  a  decision  might  be  made  on  arrival,  service  or 
both.  Work  of  Miller  [10],  Llppman  and  Rose  [8]  are  of  the  first  category. 
In  Miller  the  decision  is  made  to  reject  or  to  accept  an  arrival  which  can 
be  from  one  of  the  n  possible  types,  where  the  nth  type  customer  offers 
a  reward  Rq  .  The  queueing  system  he  considers  is  M/M/C,  where  no  queue  is 
allowed.  Llppman  and  Ross  generalize  Miller's  problem  to  a  general  arrival, 
service,  and  reward  distributions  with  an  Infinite  number  of  arrival  types 
but  only  one  server.  In  both  cases  simple  policies  are  found  in  order  to 
maximize  the  expected  average  return. 

Work  of  Crablll  [2],  McGill  [9]  and  Mitchell  [11]  are  of  the  second 
category  where  the  decisions  are  made  on  servicing  facility. 

Crablll  found  the  optimal  servicing  rates  for  an  infinite  queue.  The 
system  he  considers  is  M/M/1  and  the  policy  he  finds  is  switchover. 

McGill  considers  a  multi-channel  queue  with  changing  number  of  servers 
as  a  function  of  persons  in  the  system. 
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Mitchell,  in  the  work  still  ir.  prop.r ’.ss,  considers  a  single  server- 
queue  with  Poisson  arrivals,  where  each  customer  in  the  system  requires  a 
random  length  of  service  which  becomes  known  at  the  time  when  customer  enters 
service.  The  decision  is  made  as  to  which  of  two  rates  to  use. 

Klimov  in  [  7  ]  considers  an  m  servers  queue  with  nonstationary  Poisson 
arrivals  and  nonstationary  exponential  service.  One  of  the  cases  considered 
there  is  where  the  arrival  and  service  rate  are  not  known  but  must  be  found 
in  order  to  optimize  an  objective  function.  The  method  used  is  optimal 
control. 

1.3  Notations  and  Definitions,  Mathematical  Background 

In  the  next  chapters  we  will  formulate  our  queufeing  problem  as  a  semi- 
Markov  decision  process.  The  early  contributers  of  these  processes  are 
Jewell  (6  ]  and  Howard  [4  j,  [  5  ]  and  recently  Ross  [12],  [13].  The  notations 
and  definitions  used  here  are  of  Ross. 

Definition; 

A  semi-Markov  decision  process  is  a  process  which  is  observed  in  each 
review  point  and  is  found  to  be  in  one  of  a  possible  number  of  states.  The 
set  of  all  possible  states  is  called  the  state  space  S  .  After  observing 

the  state  an  action,  a  ,  must  be  maoe  from  a  set  of  possible  actions.  The 

« 

set  of  all  possible  actions  is  called  the  action  space  A  .  Whenever  the 
state  is  1  and  action  a  is  chosen,  then 

(I)  A  transition  to  state  j  occurs  with  the  probability  P^ta)  • 

(II)  If  the  next  state  of  the  system  is  j  ,  then  the  time  until  the 
transition  from  i  to  J  is  a  random  variable  with  distribution 

V*>  • 

(ill)  The  action  a  taken  in  state  i  will  specify  the  cost  Incurred 
in  the  next  stage  of  the  process. 
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A  policy  is  a  rule  fcr  c  loosing  actions  when  the  current  state  and  the 
past  history  of  the  orocess  is  given. 

A  stationary  policy  is  a  nonrandomized  policy  where  the  action  chosen 

9 

at  a  time  only  depends  on  the  state  of  the  process  at  that  time. 

Average  Cost  Results 

Let  Z(t)  be  the  total  cost  incurred  by  time  t  and  be  the  cost 

incurred  during  the  nth  transition  interval  and  be  the  length  of  this 

interv  :.  For  each  policy  n  and  state  i  ,  we  define 

(1.1)  ♦*(!)  -  lim  -  i] 


represents 
"nicer" 
that  under 

f  any 


Then  it  can  be  shown  (see  Ross  [12])  that 


and 


(1.2) 


$  (i)  *  lim 
rt 

n-x» 


l  yxj  - 1 

i-i  3 


I  yx!  - 1 

j»l  J 


where,  is  the  initial  state  of  the  system.  Although,  <f> 

2 

the  long-run  average  expected  cost,  it  turns  out  that  <ji  is  a 
criteria,  in  analytical  sense,  to  work  with.  It  also  turns  out 
some  reasonable  conditions  these  two  criteria  are  equal. 

Let  T  be  the  time  of  the  first  return  to  state  i  ,  and 
stationary  policy  ani  suppose  that 


(1.3) 


Ef[T/X1  -  i]  <  » 


♦  J(i)  -  ^U)  . 
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A  method  to  find  an  optimal  stationary  policy  using  $  is  given  in 
[12],  Thus,  if  (1.3)  is  satisfied  then,  this  same  policy  is  optimal  for 
Now,  let  c(i,a)  denote  the  expected  cost  incurred  in  a  transition 
interval  that  begins  with  action  a  being  taken  in  state  i  .  Also,  let 
t(i,a)  denote  the  expected  length  of  such  a  transition  interval.  It  should 
be  noted  that,  when  the  relevant  criteria  is  given  by  Equation  (1.2),  then 
without  loss  of  generality  we  may  assume  that  the  cost  incurred  in  such  an 
interval  (and  the  length  of  such  an  interval)  is  with  probability  one 
c(i,a) (x(i,a))  .  This  will  be  assumed  here  on. 

Note  that  this  implies  that  V^(i)  ,  the  minimal  total  expected  a- 
discounted  cost  starting  from  state  i  ,  ratifies 


(1.4) 


V  (i) 
a 


min 

a 


c(i,a)  +  e 


-ca(i,a) 


l  p^wyj)! 


i  >  0  . 


The  following  theorem  may  be  found  in  Ross  [13], 


Theorem : 

If  c(i,a)  is  bounded,  and  if  there  exists  an  N  <  ®  such  that 

|va(i)  -  Va(0)|  <  N  for  all  a  ,  all  i 


then,  there  exists  a  bounded  function  h(i)  and  a  constant  g  such  that 

(1.5)  h(i)  »  min  |c(i,a)  +  £  P.  (a)h(j)  -  gT(i,a)J  i  >  0  , 

a  (  j-0  ) 

* 

and  if  it  Is  a  policy  which,  for  each  i  ,  prescribes  an  action  which 
minimizes  the  right  side  of  (1.5)  then 


g  »  $2*(i)  =  "»in  $2(i) 

TT  IT 


all  i  >  0  . 


This  theorem  will  be  used  in  later  chapters. 
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Chi'J’TER  II 
DISCOUNTED  MODEL 


2.1  Introduction 

In  this  chapter  we  will  formulate  the  queueing  system  as  a  semi-Markov 
decision  process.  We  assume  a  queueing  system  with  Poisson  arrivals  at 
rate  >  ,  and  a  single  exponential  service  facility.  The  service  facility 
can  perform  its  duty  using  one  of  the  K  available  service  rates, 

_<  m  •••  .1  •  When  service  rate  is  in  effect,  the  service  time 

is  a  random  variable  which  is  exponentially  distributed  with  expected  service 
time  l/p^  •  Those  arrivals  finding  system  with  N  persons  will  not  stop 
and  will  be  lost.  In  other  words  the  queueing  facility  can  only  hold  N 
persons  at  a  time.  The  problem  will  be  formulated  in  two  cases.  First,  the 
operator  can  only  switch  at  the  time  of  departure  of  a  customer  in  the  system. 
Second,  where  the  operator  can  switch  at  either  an  arrival  or  at  a  departure 
time  of  a  customer.  These  switches,  if  made,  must  minimize  the  overall  expected 
a-discounted  cost  of  the  system.  The  following  cost  structure  is  considered: 

(i)  a  service  cost  rate  cCy^)  ,  the  cost  rate  incurred  when  service 

rate  is  in  effect.  It  is  assumed  that  cCp^)  is  nondecreasing 

In  uk  . 

(ii)  For  each  customer  served  a  reward  of  A  will  be  earned. 

In  Section  2  the  problem  will  be  formulated  as  a  semi-Markov  decision 
process  and  in  Section  3  some  properties  of  optimal  value  function  defined  in 
Section  2  will  be  studied.  In  Section  A  some  special  cases  will  be  analyzed 
and  a  sufficient  condition  for  the  existance  of  a  simple  switchover  policy 
will  be  given. 
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?_  Semi-Markov  Decision  for  tHicueing 

We  must  define  the  state  of  the  system,  the  available  actions,  and  the 
probabilistic  law  or  the  law  of  otion.  The  state  of  the  system  is  defined 
as  the  number  of  customers  in  the  queue.  The  state  space  S  ,  is  thus  a  set 
of  N  +  1  nonnegative  integers  S  =  {0,1,  . ..,  N)  .  The  action  space  K  is 
defined  as  a  set  of  K  nonnegative  integers  K  m  {1,2,  ...»  K }  ,  where 
action  k  corresponds  to  rate  . 

The  transition  probabilities ,  if  ^witching  is  only  allowed  at  the 
departure  points  can  be  easily  found  to  be  as  follows: 


xV 


!<X  +  ba) 


t+1 


t  -  0,1,  ...,  N-i-1 


(2.1)  Pj j  (a)  - 


xV 


t-N-i  (1  +  p  ) 

a 


t+1 


j  <  i  -  1  all  i>l 


J  -  i  +  t  -  1  i  >_  1 

j  -  N  -  1  i  >  1 

j  >  H  i  >_  1 

j  -  1  i  -  0  . 


If  switching  is  allowed  in  both  departure  and  arrival  points  then 


j  *  i  -  1  ,  i  +  1  i  >  1 


X  +  w. 


v*>  ■< 


X  +  p 


j  -  i  -  1 

J  -  i  +  1 
J  “  1 


i  >  1 


i  >  1 


i  -  0  . 


When  the  first  set  of  transition  probabilities  are  in  effect  then  we  will 
say  that  we  are  in  Case  1,  and  similarly  for  Case  2.  Now  for  Case  1,  if 
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Va(i)  is  the  optimal  valu*?  function  as  defined  in  Section  3  of  the  previous 
chapter,  then 


c(u  ) 


V  (i)  •  min 
a  ip 

ala 


-  —  1 
p  N-i-1  XJp 

-  A  +  e  *  l  - ^~r+1  . 

1=0  a  +  u.)J  1  a 


v  (i  +  j  -  1) 


+  e  *V  (N  -  1)  l 


j'N-i  (X  +  p  )^+1l 


i  -  1,  ....  N-l 


(2.3) 


V  (N)  *  min 
a 

a 


c(u  ) 

- 5-  -  A  +  e  aV  (N  -  1) 

U_  a 


V  (0)  -  e  AV  (1) 
a  a 


e(pa) 

where, - A  is  the  cost  incurred  in  a  transition  interval  of  length 

Pa 

1/p  ,  the  expected  transition  time  when  rate  p  is  in  effect.  In  other 

®  a 

words,  VQ(i)  is  the  minimal  expected  a-discounted  cost  starting  from  state 

i  ,  where  the  operator  can  only  switch  at  departure  points.  In  the  case 

where  switching  is  allowed  at  any  time  (that  is,  in  Case  2)  the  optimal 

value  function  V  (i)  satisfies 
a 


va(i) 


min 


c(u.)  -  U.A  .  .  X  +  ",  X 


X  +  p 


+  e 


x  tt:  V1  + 


(2.4) 


X  +  wa  Wa  - 

+  e  r+V  V1  - 1)1 

a 

)c(p  )  "  p”_ 

Vq(N)  -  min  i-  -  -  A  +  e  -  l)j 

a  v  a 


i  -  1,  N-l 


V  (0)  -  e  AV(1)  . 
a 


* 
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Of  course,  V  (i)  and  V  (i)  are  different. 
a  a 

2.3  Properties  of  the  Optim al  Value  Function 

In  this  section  some  properties  of  the  optimal  value  function  V^(i) 
will  be  studied. 


Lemma  1; 

c(u  ) 

V  (i)  >_  0  for  all  i  ,  if  and  only  if  — - — —  >_  A  for  all  a  . 

wa 

Proof : 

The  proof  is  by  induction,  define 


(2.5) 


Vo(l.l) 


min 


c(w  ) 

a 


-  A 


and  recursively 


)c(u  )  N-i-1  **  u  XJu 

V^U.n)  -  min  <— - A  +  J  e  - rrr  V  (i  +  j  -  l,n  -  1) 

a  I  Ma  j»0  (X  +  u  )^ 


(2.6) 


a 

u 


*v 


+  e  av  (N  -  l,n  -  1)  l  ....  . 

°  J-N-i  (X  +  p.)J+1( 


Functions  1^(1, n)  will  converge  to  Va(i)  as  n  goes  to  infinity,  see  [13]. 

c(u  ) 

(i)  If  -  >  A  for  all  a  ,  then  clearly  V  (i,l)  >  0  . 

wa  "  “ 

Now  assume  Va(i,m)  >_  0  for  all  m  <_  n  -  1  .  Then,  by  induction  assumption 

all  terms  in  the  right  side  of  (2.6)  are  nonnegativo.  Hence,  V^(i,n)  _>  0 

for  all  n  ,  and  in  the  limit  V  (i)  >  0  . 

a  - 

c(w  ) 

(ii)  If  VQ(i)  >_  0  ,  for  all  i  then  - — 


>  A  for  all  a  . 
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c^k> 

Assume  the  contrary,  that  -  <  A  for  some  k  ,  then  V(i.l)  +  t  <  0 

u. 

k 

for  some  c  >  0  ,  and  using  this  rate  In  all  states  implies 

V  (i,n)  <  -e  for  all  i  and  n  ,  or  in  toe  limit  V  (i)  <  G  which  contradicts 
a  a 

the  assumption. 

Q.E.D. 

e(wk) 

An  immediate  conclusion  of  this  lemma  is  if  -  <  A  for  some  k 

Wk 

then  V  (i)  <  0  for  all  i  . 

Lemma  2: 

c(y  ) 

V  (i)  is  nonincreasing  if  and  only  if  -  <  A  for  some  a  . 

wa 

Proof ; 

We  prove  this  lemma  by  induction  as  of  the  previous  lemma. 
c(u  ) 

(i)  jf  -  <  A  then  V  (1)  is  nonincreasing. 

y#  o 

From  Equation  (2.5) 


V  (1,1)  -  V  (1  +  1,1)  for  all  1  >  1 

U  a  — 

and 

-4  +  or) 

Va(0,l)  -  e  '  W*'v(l,l) 

*  (c(wa)  ) 

where,  y#  is  the  rate  which  minimizes  j— - A>  .  By  assumption 

V(l,l)  <  0  .  Hence,  Va(0,l)  >  V(l,l)  »  V(i,l)  i  >  1  and  the  lemma  is 

true  for  n  ■  1  .  Now  assume  the  lemma  is  true  for  all  m  <  n  -  1  ,  then 


Lemma  3: 

c(u  ) 

Va(i)  is  nondecreasing  if  and  only  if  — -  A  for  all  a  . 

Proof : 

Proof  of  this  lemma  is  similar  to  the  previous  pne,  and  is  omitted 


here. 


13 


Remark: 

Lemmas  1,  2,  and  3  can  be  proved  for  functions  V^(i)  .  The  proof  is 
similar  to  those  lemmas  and  is  omitted  here. 

The  following  lemma  will  be  useful  in  the  next  chapter. 

Lemma  4: 

(a)  There  exists  an  M  <  «  such  that 

|va<i)  -  Vft(0)|  <  H  for  all  a  ,  and  all  i  . 

(b)  There  exists  an  M  <  ®  such  that 

|Va(i)  -  VO)  I  <  M  for  a11  «  •  and  a11  1  • 

Proof: 

(a)  Let  (rlf  ...,  rN>  be  the  optimal  service  rates  used  in  states 

(1,  ...,  N)  then  the  embeded  Markov  chain  corresponding  to  departure  points 

Is  Irreducible  and  positive  recurrent,  see  (2.1).  Let  be  the  expected 

number  of  transition  starting  from  state  i  before  reaching  state  0  ,  then 

K.  <  «  for  all  i  .  Let  K  ■  max  K.  ,  for  the  case  V  (i)  <  0 
x  .1  a  — 


The  left  side  of  the  above  inequality  is  also  bounded  by  0  .  Hence 


Iva(i)  -  Va(0)|  <  K  min 

a 
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for  the  case  V  (1)  >  0 
a  — 


V  (i)  -  V  (0)  <  K  wax 
a  a  - 

a 


f  c(tla> 
(  % 


A 


or 


lVa(i)  "  V0)l  1  K  max 

a 


& 


Now  let 


M  ■  max 


;  K  max 
a 


(  “a 


then 


(2.9) 


|  va ( 1 )  -  VJO)  |  <  M  . 


(b)  The  proof  of  this  part  is  similar  to  the  first  part  using  the 
transition  probabilities  (2.2),  and  arguing  in  the  same  way. 


Q.E.D. 


2.4  Some  Properties  of  Optimal  Policy 

In  this  section  some  special  cases  will  be  analyzed  and  a  conditional 

theorem  for  the  existance  of  a  switchover  policy  will  be  given.  We  define 

A 

f  (1)  to  be  the  optimal  action  at  state  i  .  The  following  lemmas  and 
theorems  are  proved  for  Case  1. 

Lemma  5: 

*  c(wa)  * 

If  f  (N)  ■  k  and  A  >  -  for  some  a  ,  thfen  f  (i)  £  k  for  all 

^a 

i  £  N  ,  in  other  words  the  largest  rate  is  used  in  the  largest  state. 
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Proof: 


By  assumption 


C<V 


0 

Pt. 


-  A  +  e 


V  (N 
a 


-  1)  < 


e(wa> 


-  v. 


A  +  e  *V  (N  -  1) 
a 


all  aO 


Let  g(i,a,k)  be  the  difference  between  two  expected  future  costs  when 
action  a  or  k  is  taken  In  1  ,  then 


(2.10)  g(N,a,k)  - 


c(pa)  c(wk) 


ya  yk 


♦  (/  “•  - u‘) 


e  /V(N  -  1)  . 


By  assumption  g(N,a,k)  0  for  all  a  k  .  Assume  in  the  contrary  that 
there  exists  state  1  such  that  f*(i)  •  a  >  k  then 


c (u  )  c(p.)  N-l-1 

•<*.*.»  l 

ya  yk  J-O 


X^U 


-  —  1 

V  XJu, 

a  _  a  k 

6 


+  u  )3+1 


(X  +  W  ) 

Q 


(X  +  wk)J+1 


(2.12) 


V  (1  +  j  -  1)  +  V  (N  -  1)  l 

J-N-i 


Xju 


a 

u. 


(X  +  ua) 


J+l 


XJW 


(X  ♦  ,k)1+1 


k  Mk 


M.  yW 

where  g(l,a,k)  <  0  .  In  (2.10)  substitute  for  e  and  e  the 

following: 


a  _ a_ 

y  y  • 

e  *  -  e  *  V  - ^ _ 

L  44.1  » 


xV 


xV 


Wk  Wk  r 

t  j+l  »  e  I  - ^-TTT 

j-o  (x  +  y.)J  j-o  ( \  +  w.  )j+1 


*• 
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then 


g(l,a,k)  -  g(N,a,k) 


(2.13) 


N-i-1 

l 

J-o 


xK 


-  —  i 

U  X  Mt 


a 

Mu 


(x  +  u  )J+1 


(X  +  Mk) 


J+l 


(V  (1  +  j  -  1)  -  V  (N  -  1)]  . 
a  a 


For  nonincreasing  Va(i)  ,  Vfl(i  +  j  -  1)  -  Vfl(N  -  1)  is  nonincreasing  and 
nonnegative.  And  the  function  h(j)  defined  as 


h(j)  - 


X^M 


-  —  1 

M  XJy, 

a  k 


q 

M,. 


a  + »,) 


j+i 


(X  +  M^) 


j+1 


a  >  k 


changes  sign  from  positive  to  negative  (note  that,  function 

_  a 
x 


S(x)  - 


A 


J+l 


(X  +  x)- 

and  decreasing  for  j  >  J)  and  since 


is  increasing  for  j  <_  J  ,  some  J  ,  0  <  J  <_  N 


a  _ a_ 

•  M  Mu 

l  h(j)  »  e  a  -  e  >0 

J-o 

then  the  positive  terms  in  (2.13)  dominate  the  negative  terms.  Hence, 

g(i,a,k)  -  g(N,a,k)  >_  0 
g(l,a,k)  >_  g(N ,a,k)  _>  0 

* 

which  contradicts  the  assumption  that  f  (i)  ■  a  . 


Q.E.D. 


Lemma  6: 

Let  there  exist  an  I  such  that  the  action  space  K  -  (1,2,  ...,  K) 
can  be  partitioned  into  two  subsets  ■  (1,2,  ...,  1}  and 
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C(wa}  (  W«  ) 

(2.15)  V  <N)  -  V  (N  -  1)  -  — -  A  +  \e  8  -  l/V(N  -  1)  . 

wa 

c(u  ) 

By  Lemma  2  V  (N)  -  V  (N  -  1)  <  0  and  by  assumption - A  >  0  ,  since 

a  a  u  — 

_  J5. 

e  -  1  <  0  ,  V(N  -  1)  <  0  then,  (2.15)  is  impossible.  Therefore,  a 
cannot  be  an  element  of  . 

Q.E.D. 

Theorem  1: 

*  * 

If  N  ■  2  then  f  (1)  <_  f  (2)  ,  i.e.,  the  optimal  policy  is  switchover. 
Proof: 

The  proof  immediately  follows  from  Lemma  5. 

Theorem  2: 

c(va) 

If  N  -  3  and  y  <  A  ,  and  -  <  A  for  some  a  ,  then  the  optimal 

K  P_ 


policy  is  switchover,  i.e., 
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Proof : 

By  Lemma 
f*(l)  £  f*(2) 
Assume  f 

g(l,k,a) 


By  assumption 

g(2,k,a) 


£*( 1)  <  f*(2)  <  f* (3)  . 


5  f  (1)  <  f  (3)  and  f  (2)  <  f  (3)  .  We  only  need  to  sh6w 


(1)  ■  k  ,  then  for  any  a  <  k 


c(pk)  c  (m^) 


X  +  y 


a 

k  Wk 

e 


a  a 

e 


X  +  y 


V°> 


Xy. 


a 

Uu 


Xy 


a' 

a 

- r  e 


(A  +  V 


(A  +  y  )' 
a 


vod) 


+  V(2)  l 
J-2 


xjy. 


-  —  i 

pk  X  ua 


(A  +  „k>3+1 


(X  +  y  )j+1 


g(l,k,a)  <  0  .  We  now  show  that  g(2,k,a)  <_  g(l,k,a)  <_  0 


"k  "k 


A  +  y. 


A-  ““I 

+  va  J“ 


(1) 


Auk  «k  Ay  y 

- -  e - r  e 


(A  +  yk)' 


(A  +  y,)' 


V  (2) 

a 


+Va(2)  l 
j-2 


X^y, 


a 


Xy. 


a 

y. 


(X  +  yk) 


J+1 


(X  +  yfl) 


J+1 
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g(2,k,a)  -  g(l  ,k,a)  -  e  -  e 

k  a 


fr e  lv0(1)  -  V0)) 


x“k  ‘-k 

2  e 

(X  +  Pk)  (X 


c  * 

Xy*  ” 

&  a 

+  U  )2  6 

rn  J 


[V  (2)  -  V  (1)J 
a  a 


since  V(l)  <  V(0)  and  V(2)  <  V(l)  and 


Pk  *k  *a  ,  ya  n 

rrir  e  "  ITT  e  >  0 

k  a 


Xuk  “k  Xu.  „ 

- re - re  >0 

(X  +  VJ^)  (X  +  p#) 


by  assumption.  Hence, 

g(2,k,a)  -  g(l,k,a)  <_  0  . 

Therefore,  if  action  k  is  better  than  action  a  in  state  1  (for  k  >  a)  , 
then  action  k  is  better  than  action  a  in  state  2  . 

Q.E.D. 


Theorem  3: 


Assume  K.  -  2  and  A  > 


c(va) 


for  at  least  one  a  ,  and  V  (i  +  1)  -  V  (i) 

<*  a 


Is  nondecreasing  in  1  ,  then  switchover  policy  is  optimal.  That  is,  there 
exists  a  i  .ate  J  ,  0  ^  J  N  ,  such  that 


f*(i)  -  1 

for  i  .<  J 

*  .  . 

f  (i)  -  2 

for  i  >  J 

Proof: 


Assume  1  is  a  state  such  that  f  (i)  -  2  ,  then 


c(w2) 


-A+  l 


N-i-1  XJy, 


J-0  (X  +  u2) 


e  S  (i  +  J  -  1)  + 


J+l  a 


r  X  W2  u2  C(wl) 

+  l  - —  e  v  (N  -  1)  <  - -  -  A  + 

j-N-i  (X  +  V^+1  a  »1 


N-i-1  XV 

l  - L 

J-0  (X  +  Vj) 


j+l  ~  a 


1  r  A  W1 

V  (i  +  J  -  1)  +  l  - Stt  e  V  (N  -  1) 


J-N-i  (X  +  M1)J+i 


8(i,2,l) 


,)  c(m1)  N-i-1  j  XJm2  "  "  XJu  ‘  ^L 

wi  j-o  [(x  +  w2)J+1  *  (x  +  Ml)J+1  * 

[J  -  i  .  -2-1 

X  W2  *V  U, 

(X  +  w2)J+1  (X  +  Wl)J+1 


V  (N  -  1)  <  0  . 
a 


We  now  show  that  g(i  +  1,2,1)  <  0  which  impliea  f*(i  +  1)  -  2  . 


g(i  +  1,2,1)  - 


c(u2)  c(iij)  N-i-2  f  XJ 


- l  - 1 - -  .  2 

W1  J-0  [(X  +  y2)j-fl 


p  XJy? 

V*  + +  I  - St  • 

J-N-i-1  [(X  +  y2)J+1 


Va(N  -  1)  . 


_ “’I 

a  +  Ul>3+1  J 

_ iV  - ^1 

»  *  U1)J+1  J 
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g(i  +  1,2,1)  -  g(i,2,l) 


N-i-1 

I 

j=0 


A^y, 


A^y. 


(2.16) 


(A  +  y2)J+1 


(A  +  y^) 


j+1 


[va(i  +  j)  -  va(i  +  j  -  1)]  . 


Now,  since 


h(j) 


\K. 


(A  +  V'2) 


.1-^1 


a 


aV 


(»  +  w1)3+1 


a 


changes  sign  from  positive  to  negative,  £  h(j)  >  0  ,  and  V  (i  +  j)  - 

j-0  “ 

v0,(1  +  J  "  1)  <0  then  the  negative  terms  in  the  right  side  of  (2.16)  dominates 
the  positive  terms.  Hence 

g(i  +  1,2,1)  -  g(i,2,l)  <  0 
g(i  +  1,2,1)  <_  g (i ,2 ,1)  <  0 

or  f*(i  +  1)  -  2  . 


Q.E .D. 
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CHAPTER  III 

AVERAGE  COST  MODEL, TWO  SERVICE  RATES 

3.1  Introduction 

In  this  chapter  we  will  consider  the  same  queueing  system  with  the 
same  cost  structure,  but  the  criterion  for  decision  rules  will  be  to  minimize 
the  long  run  expected  average  cost.  We  assume  that  only  two  service  rates 
p^  <  p^  are  available  and  the  service  rate  can  be  switched  either  at  an 
arrival  or  departure  epoch  during  the  process.  When  service  rate  p  is  in 
effect,  the  service  time  is  a  random  variable  which  is  exponentially  distributed 
with  mean  l/pfl  .  We  will  use  methods  of  Ross  as  stated  in  the  last  chapter 
to  find  the  form  of  optimal  policy. 

In  Section  2  the  existence  of  stationary  policy  will  be  proved,  and  in 
Section  3  the  form  of  the  optimal  stationary  policy  will  be  found.  An 
efficient  computational  method  will  be  given  in  Section  A.  In  Section  5 
the  problem  of  "hesitating  customers,"  who  join  the  queue  with  certain 
probability,  is  analyzed  and  simple  optimal  policy  is  found. 

3.2  Existence  of  Stationary  Policy 

We  use  the  method  of  semi-Markov  decision  processes  to  analyze  the 
queueing  problem.  The  state  of  the  system,  as  in  Chapter  II,  will  be  the 

c 

number  of  customers  in  the  system  and  the  action  space  is  K  ■  {1,2}  .  The 
transition  probabilities  are  the  same  as  those  given  in  (2.2). 

As  was  stated  in  Chapter  I,  under  the  following  three  conditions  there 
exists  an  optimal  stationary  policy. 

(1)  c(l,a)  ,  the  expected  cost  during  a  transition  Interval,  is 
bounded. 

(ii)  |Va(i)  “  Vq(0)|  <  M  for  all  i  and  a  and  for  some  M  <  «  . 

V  (i)  are  as  defined  in  Chapter  II. 
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(iii)  E.[T/Xj  =  t]  v  •  ,  where  T  is  the  time  of  the  first  return  to 
state  i  ,  is  the  initial  state  of  the  system .  ;  nd  f  is 

a  stationary  policy. 

• 

The  optimal  stationary  policy  can  be  found  from  the  optimality  conditions 
stated  in  Chapter  I,  and  will  be  discussed  later  in  this  section.  Under 
conditions  (i) ,  (ii) ,  and  (iii)  this  optimal  policy  will  minimize  $*(i)  , 
the  expected  long  run  average  cost  starting  from  state  i  ,  as  defined  in 
Chapter  1. 

Lemma  1: 

Conditions  (i) ,  (ii)  and  (iii)  hold  for  the  queueing  problem. 

Proof : 

The  proof  of  (i)  is  immediate,  (ii)  was  proved  in  Chapter  II  (Lemma  4). 
Since  the  chain,  defined  in  part  b  of  Lemma  4,  Chapter  II,  is  irreducible 
and  positive  recurrent  then  the  number  of  transitions  before  the  first 
return  to  i  is  bounded  for  all  i  ,  and  so  is  the  expected  time  of  each 
transition.  Hence  (iii)  holds. 

Q.E.D. 


3.3  Derivation  of  Optimal  Policy 

In  this  section  we  first  modify  the  optimality  conditions  for  the 
queueing  system,  and  then  the  form  of  optimal  policy  will  be  derived. 

The  optimal  stationary  policy  is  one  which  prescribes  the  minimizing 
action  in  the  following  equations. 


h(i)  -  min  Jc(i,a)  +  ]>  P..(a)h(J) 

a  I  j  J 


gx(i ,a) 


I 

i 


Plj (a)  were  defined  in  (2.2)  and 


c(0,a)  “  0 


c(i,a) 


-  4 

+  m ) 


1-1,  ....  N 


Hence , 


T(0,.)  -  f 


t(l,»)  -  1-1,  ....  H 


(3.1) 


h(l)  -  f 

’ 

c(y  )  -  jj  A  .  v 

min  - - ; - —  +  - -  h(i  +  1)  +  - /■—  h(i  -  1)  -  -r-^ - 

\  +  w  atU  AXU  A-tU 

a  a  a  a  a 

i  -  1 . N-l 

|c(p  )  -  U  A  V  .  ) 

- f-j - —  +  t-t* —  h(N  -  1)  +  - h(N)  -  - {  . 

i  +  u.  *  +  x  +  *.  X  +  UM 


(3.1)  can  be  written  aa  follows: 


fi  -  Mh(l)  -  h(0)] 


c(ua)  “ 

X  +  u 


+  FTT  lh(i  +  1} 


-  h(i)l  -  [h(i)  -  h(i  -  1)] 


x  +  u 


c(w#)  ■  WaA 


i  -  1,  N-l 


x  +  u  x  +  u 


-  x  *  (h(N)  -  h(N  -  1)]  - 


X  ♦  W. 


or 


26 


*• 


27 


Hence,  3^  is  maximized  it  for  all  i  ,  and  minimized  if 

for  all  i  . 

Let  be  the  effective  departure  rate  or  long  run  average  departure 

rate,  then  *  X(1  -  $^)  .  Since  c(r^)  is  the  cost  rate  when  r^  is 
used  then 


is  the  average  service  cost  rate.  The  average  return  rate  will  be: 


N 

G  -  1(1  -  £L)A  -  l  B.c(r  )  . 

i-1 


Zti 

N 

Now  since  £  B.r  is  the  long  run  avot'ge  service  rate  or  the  effective 

i-1 

service  rate,  then 


N 

l  v,  -  mi  -  <y  • 

i-1 


Hence , 


(3.3) 


g  <  max  A(1  -  fO 
—  U 


rl  •  •  •  •  <r(j 

VW 


c(uR) 

But  by  assumption  A - >_  0  then,  the  right  side  of  (3.3)  is  maximized 

MK 

if  0N  is  minimized.  8^  is  minimized  if  r^  -  for  all  i  .  Therefore 
for  this  choice  of  rates  the  right  side  of  (3.3)  is  maximum,  but  for  the 
same  choice  of  rates  we  will  have  equality  in  (3.3). 


(b) 


8 


max 

rl»*‘ 4  »rj 
Titu 


N 

A(1  -  8n)A  -  [  61c(r1) 


max 

r^,«  •  •  ,rj| 

r1eu 


A(1  - 


6n)A- 


c(p1) 


N 

l  8^ 

i-1  x  1 


Q.E.D 


by  assumption  or 


(3.4) 
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c(y.,) 

Since  A - <  0  then  the  right  side  of  (3.4)  is  maximized  if  S  is 

N 

maximized,  but  8^  is  maximized  if  *  y^  for  all  i  ,  and  for  this 
choice  of  rates  we  have  equality  in  (3.4). 

Q.E.D. 


Lemma  3 : 


Let  R 


c(y2)  -  c(y1) 
y2  "  P1 


-  A  .  Then,  the  optimal  policy  is  to  use  y^  in 


state  i  if  and  only  if 


(3.5) 


h(i)  -  h(i  -  1)  <  R  . 


Proof : 

*  * 

Let  f  (i)  be  the  optimal  action  when  in  state  i  ,  then  f  (i)  =  1 

if  and  only  if 


(3.6) 


c(y:)  -  yxA  +  A  ( h  (i  +  1)  -  h(i)  ]  -  y^hU)  -  h(i  -  1)]  < 
c(y2)  -  y2A  +  X(h(i  +  1)  -  h(i) )  -  y2(h(i)  -  h(i  -  1)] 


or 


c(y,)  -  c(y.) 

h(i)  -  h(i  -  1)  <  - - - —  -  A 

-  w2  -  Vl 


Q.E.D. 


Theorem  1: 

c(ya) 

If  A  >_ -  for  at  least  one  of  the  rates,  then  there  exists  a 

Wa 

state  J  »  0  i  J  £  N  ,  such  that 

f*(i)  -  1  i  -  1,2„  ... ,  J-l 

f*(i)  -  2  i  -  J,J  +  1,  ... ,  N  . 


That  is,  the  optimal  policy  is  switchover. 
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Proof: 

*  * 

Let  k  be  the  smallest  statu  such  that  f  (k)  ■  2  and  f  (k  -  1) 
We  consider  two  cases: 


Case  1: 

k  «  1  ,  then 

(3.7)  h(0)  =  h(l)  -  & 

(3.8)  g  -  X [h (1)  -  h(0)J  >  XR 

•k 

by  Lemma  3.  For  state  1  ,  since  f  (1)  =  2 

g  -  c(y2)  -  w2A  +  X [ h ( 2)  -  h(l) ]  -  y2[h(l)  -  h(0)] 

(3.9) 

g  <  c(y2)  -  y2A  +  X[h(2)  -  h (1) ]  -  y2R  . 

From  (3.8)  and  (3.9) 


AR  <  c(y2)  -  p2A  +  A[h(2)  -  h(l) ]  -  y2R 


or 


(3.10)  XR  +  y2R  -  c(y2)  +  m2A  <  X[h(2)  -  h(l) ]  . 


cCyj^)  c(u2) 

Now  for  -  -  we  have  y_R  -  c(y,)  +  y,A  >_  0  since 

Pi  Po  *  £.  L 


c(y2)  -  c(u1) 

y2R  -  c(y2)  +  p2A  -  y2  - - — — - - y2A  -  c(y2)  +  y2A 


2  H1 


c(y-)  - 


c(y.) 


T  n  -  u 

u2  U1  1 


VP  m11u1C<1,2)  '  ',2C(U1> J  i° 
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Hence,  from  (3.10) 


h(2)  -  h(l)  >  R 


or  by  Lemma  3,  f  (2)  *  2  .  Continuing  in  this  manner  we  can  show  f  (i) 

c(u2)  c(w1> 

for  i  =  3, A . N  .  But  for  -  < - by  Lemma  2(a)  we  know  y_ 

WWW  lili  2 

2  1 

optimal  in  all  states.  This  completes  the  proof  for  Case  1. 


-  2 
is 


Case  2: 

k  >_  2  .  By  Lemma  3 

|h(k  -  1)  -  h(k  -  2)  <  R  f*(k  -  1)  -  1 

* 

h(k)  -  h(k  -  1)  >  R  f  (k)  -  2 

and  for  state  k  and  k  -  1  , 

(g  -  c(ua)  -  wxA  +  X[h(k)  -  h(k  -  1)]  -  Wl[h(k  -  1)  -  h(k  -  2)] 
(g  -  c(u2)  -  u2A  +  A [h(k  +  1)  -  h(k)]  -  u2fh(k)  -  h(k  -  1)] 


or  by  (3.11) 


g  >  c(u1)  -  y jA  +  XR  -  y^R 
g  <  c(y2)  -  y2A  -  w2R  +  X[h(k  +  1)  -  h(k)]  . 

Then, 


c(y1)  -  PjA  +  XR  -  y^R  <  c(y2)  -  y2A  -  y2R  +  X(h(k  +  1)  -  h(k)] 


or 


X[h(k  +  1)  -  h(k) )  <  c(y1)  -  c(y2>  -  (yx  -  y2)A  +  (y2  -  y^R  +  XR  . 


But 


» 


M 


(n2  -  -  I  <  (i  J  •  -  t‘j)R  ■ 


H.ince , 


h(k  1  i)  -  li'vM  K 

*  * 

or  f  (k  +  1)  e  2  .  Continuing  in  the  saint;  manner  we  can  show  r  U/  -  l 

for  i  «  k  +  2,  ...»  N  which  coirplt-tes  the  proof  of  the  theorem 

Theorem  2 : 

c(u  ) 

If  A  < -  lor  a  -  1,2  .  Then,  there  exists  a  state  )  , 

“a 

0  <  J  N  ,  such  that 


* 


f  (i)  -  2 

for 

i  -  1,  ...,.) 

An  -  i 

for 

1  •  J,J  +  1,  . 

That  is,  the  optimal  policy  is  a  reverse  switchover. 

Proof : 

The  proof  is  similar  to  the  proof  of  Theorem  1.  Let  k  be  the  smallest 

*  * 

state  such  that  f  (k)  *•  1  and  f  (k  -  1)  •  2  ,  consider  two  cases; 

Case  1: 

k  -  1  ,  f*(l)  -  1  and 

h(l)  -  h(0)  <_  R  . 

For  state  0  , 

(3.12)  g  -  A[h(l)  -  h(0)  ]  <  AR  . 


For  state  1  , 


(3.13) 


g  ■  c (y -  y^A  +  A [h (2)  -  h (1) ]  -  y ^ [h ( 1 >  -  h(0)] 
g  2L  cC^)  -  WjA  +  X[h(2)  -  h (1) )  -  y^R  . 

From  (3.12)  and  (3.13) 

c(yx)  -  yxA  +  X  [ h  ( 2 )  -  h(l)  ]  -  V]K  <  XR 

(3.1A) 

X{h(2)  -  h(l)  ]  <  XR  +  y:R  -  c(yp  +  y^  . 


But 


c(y2) 


Then  ( 3 . 14)  reduces  to 

h(2)  -  h(l)  <  R 

*  * 

or  f  (2)  «  1  .  Continuing  in  the  same  manner  we  can  show  f  (i)  -  1  for 

c(y1)  c(y^) 

i  ■  3, A,  ....  N  .  If  — - -  <  — - -  then  by  Lemma  2(b)  the  optimal  policy 

is  to  use  y^  in  all  states,  which  completes  the  proof  of  Case  1. 

Case  2: 

The  proof  is  similar  to  the  proof  of  Case  2,  Theorem  1  and  details  are 
omitted  here. 

3.  A  A  Computational  Approach 

In  the  last  section  we  proved  that  the  optimal  policy  has  a  simple  form. 

e(ua) 

It  uses  (in  the  case  A  >  -  for  some  a)  y.  up  to  a  state  J  -  1  and 

ua 

then  it  is  switched  to  rate  y^  .  We  give  a  simple  computation  method  to  find 


c(y2)  -  c(y^) 

MjR  -  c(yj)  +  y jA  -  yj^  - - — — - y^  -  c(y1)  +  y^A 


“ — — —  [y.c(u9)  -  y.c(y.)]  <_  0  if 

V  “  M  A  J- 


c(y1) 


J 
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* 


Let  P^  be  the  proportion  of  time  the  system  spends  using  rate  y^  in 
a  given  switchover  policy.  Since  the  cost  rate  is  c(y^)  -  y^A  and 
c(y2)  ~  ^or  rates  and  w 2  respectively  then 

(3.15)  g  ■*  P1[c(w1)  -  UjA]  +  (1  -  P1)[c(y2)  -  p2a1 

where  g  is  the  average  rate  of  return  as  defined  before.  Now  let  P^  be 
the  proportion  of  time  that  the  system  spends  using  y^  where  state  j  is 
the  switching  state,  then 


«■}  -  Y  ej 

1  i-1  1 


where  8^  are  the  stationary  probability  for  this  given  policy 


„  A_  J 

1  i  60 


i  <  j  -  1 


.3  .  >3-1  «J 

1  j-y-j+i  0 

W1  y2 


i  I  j  i-1 . N 


-  0  i  >  N 


« 


X  XJ_1  XJ  XN  ■ 

1  +  M,  +  •••  +  J-i  +  J-1.  +  •••  +  j-iN-j+1 


U1  U2 


yl  W2 


We  first  compute 


P1  P2 


.  P 


N 

1  * 


c(Uj)  -  u^A  <_  c(y2)  -  y2A 


Now  If 
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then  in  (3.13)  we  take  as  large  as  possible,  and  J  is  found  from 


P 


J 

1 


And  if 


c(Wi>  -  PjA  >  c(u2)  _  M 2A 
then  the  optimal  J  is  found  from 


c(p#) 

The  same  method  can  be  used  in  the  case  A  <  -  ,  a  **  1,2  ,  except 

Wa 

the  switching  is  from  rate  to  rate  • 

3.5  The  Hesitating  Customers  Problem 

In  this  section  we  assume  that  the  arrival  rate  depends  on  the  state  of 
the  system.  It  is  assumed  the  arrival  rate  is  AP^  when  the  system  is  in 
state  i  ,  where  0  <  <  1  .  In  other  words  an  arriving  customer  will 

Join  the  queue  with  a  probability  P^^  whenever  the  system  is  in  state  i  . 

The  same  cost  structure  is  considered  and  the  service  facility  will  be  the 
same  as  in  the  last  sections.  Let  K(i)  be  the  relative  cost  value  corresponding 
to  state  i  ,  equivalent  to  h(i)  in  the  last  sections,  then  the  optimal 
stationary  policy  could  be  found  from  the  following  set  of  equations: 

/g  -  AP0(H(i)  -  H(0) ] 

g  -  min  {c(p  )  -  W  A  +  AP.[H(i  +  1)  -  H(i)J  -  p  [H(i)  -  H(i  -  1))} 

8  8  X  8 

(3.16)  < 

I  1-1 . N_1 

(g  -  min  {c(p  )  -  p  A  -  p[H(N)  -  H(N  -  1)]}  . 

\  a  a  a 
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With  this  definition  of  relative  cost  functions  one  can  state  the  equivalent 
of  Lemma  3  as  follows : 

Lemma  3* : 

c(y-)  -  c(vO  * 

Let  R  -  - - - = - A  then  f  (i)  -  1  if  and  only  if 

w2  '  U1 

H(i)  -  H(i  -  1)  <  R  . 

The  proof  is  immediate. 

Lemma  A: 

c(u  ) 

If  A  >  - =-  for  some  a  and  H(0)  -  0  ,  then  H(i)  is  nonincreasing 

"  "• 

as  a  function  of  i  . 

Proof: 

(3.17)  g  -  AP0[H(1)  -  H(0) ]  -  XP0H(1)  . 

c(u  ) 

But  A  >_ -  then,  g  <_  0  and  H(l)  <_  0  .  Now  it  can  easily  be  shown  that 

Wa 

V  (i)  as  defined  in  Chapter  II  is  nonincreasing  for  the  hesitating  customers 
a 

problem  and  since  H(l)  has  the  same  structural  form  as  Vft(i)  (see  Ross  [13]) 
then  H<1)  will  be  nonincreasing. 

Q .  E .  D . 

Theorem  3: 

c(p4) 

If  the  P^’s  are  increasing  in  1  and  -  <  A  for  at  least  one  a 

Ma 

c(w.)  c(y.) 

and  -  <_ -  ,  then  the  switchover  policy  is  optimal  for  the  hesitating 

M1  W2 


customers  problem. 


37 


Proof; 

*  * 

Let  k  be  the  smallest  state  such  that  f  (k)  *  2  and  f  (k  -  1)  *  1  , 
we  prove  the  theorem  in  two  cases  as  we  did  in  the  proof  of  Theorem  1. 

Case  1; 

k  -  1  . 

(3.18)  g  -  AP0[H(1)  -  h(0)]  >  APqR 

by  Lemma  3'.  Note  that  since  g  <  0  then  R  <  0  .  For  state  1 
g  -  c(p2)  -  u2A  +  AP1[H(2)  -  H(l)  J  -  p2[H(1)  -  H(0)  ] 

(3.19) 

<  c(w2)  -  p2A  +  AP1[H(2)  -  H(l) ]  -  p2R  . 

From  (3.18)  and  (3.19)  we  have 

c(w2)  -  w2A  +  AP1(H(2)  -  11(1) ]  -  u2R  >  APqR 

or 

(3.20)  AP1[H(2)  -  H ( 1)  ]  >  APqR  +  u2 R  -  c(u2)  +  M2A  . 

But  by  assunption 


P2R  -  c(w2)  +  p2A  -  —  _  ^  [c(p2)  -  c<mx)  ]  -  w2A  -  c (y 2 )  +  w2A 


w2  - 


{pjCCuj)  -  y2c(y1))  >_  0 


Hence,  (3.20)  reduces  to 


H(2)  -  H(l)  >  ~  R  -L  R  . 
1 
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0  * 

.Since  R  <  0  and  —  <_  1  ,  or  f  (2)  **  2  .  Continuing  in  the  same  way  it 
*  1 

can  be  shown  f  (i)  *  2  for  all  i  >_  2  . 

Case  2 : 

k  2  ,  then 

H(k  -  1)  -  H(k  -  2)  _<  R 

(3.21) 

H(k)  -  H(k)  >  R 

and  R  0  . 

For  state  k  -  1  and  k  we  have 

g  -  c(m1)  -  pxA  +  Pk  lX[H(k)  -  H(k  -  1)]  -  Wl[H(k  -  1)  -  H(k  -  2)] 

g  -  c(u2)  -  m2A  +  PkX[H(k  +  1)  -  H(k) J  -  vi 2 [ H (k)  -  H(k  -  1)] 

or  by  (3.21) 

g  >  cCi^)  -  WjA  +  Pk]XR  -  WjR 

g  <  c(w2)  -  U2A  +  XPk[H(k  +  1)  -  H(k)]  -  w2R 

or 

c(u2)  -  u2A  +  XPk(H(k  +  1)  -  H(k)  ]  -  w2R  >  c(Wl)  -  Wl A  +  XPk_jR  -  uf 

< 

or 

XPk[H(k  +  1)  -  H(k) ]  >  XPklR 

Pk-1 

H(k  +  1)  -  H(k)  >  R  >  R  . 

a 

Pk-1  * 

Since  R  <_  0  and  -= —  <_  1  .  Therefore,  f  (k  +  1)  ■  2  .  Continuing  in 
the  same  manner  it  can  be  shown  f  (i)  -  2  for  all  i  >  k  +  1  . 


Q.E.D. 
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r(uaJ 

Now,  let  us  assume  that  A  <  -  for  a  *  1,2  .  Then  Lemma  3'  still 

hold.  Equivalent  of  Lemma  4  can  be  stated  as  follows  and  the  proof  is 
similar  to  the  proof  of  Lemma  4. 


Lemma  4' 


e(pa) 

If  A  <  — -  for  a  *  1,2  ,  then  H(i)  is  nondecreasing. 


Theorem  4: 


c(lia) 


If  the  P^s  are  increasing  in  i  ,  A  <  -  for  a  *  1,2  ,  and 

^a 

c  (w  2 )  c(p.) 

— -  <_  — - -  ,  then  the  optimal  policy  for  the  hesitating  customers  problem 

is  reverse  switchover.  Proof  of  this  theorem  is  similar  to  the  proof  of 


Theorems  2  and  3.  Details  are  omitted  here. 
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CHAPTER  IV 

AVERAGE  COST  MODEL,  K  SERVICE  RATES 

4.1  Introduction 

In  this  chapter  the  results  of  Chapter  III  will  be  generalized  for  the 
K  service  rates,  p^  <_  p^  <_  ...  <_  p^  .  The  cost  structure  Is  the  same,  i.e., 
c(p^)  Is  the  cost  rate  of  using  service  rate  p^  .  It  is  assumed  that  c(*) 
is  a  nondecreasing  function.  A  reward  of  A  is  earned  for  each  customer 
served.  Again  the  method  of  semi-Markov  decision  processes  is  employed  in 
order  to  minimize  the  long  run  expected  average  cost.  The  state  of  the  system 
is  the  number  of  customers  in  the  system  and  the  action  space  is 
K  *=  {1,2,  ...,  K}  .  The  arrival  stream  as  before  is  the  Poisson  process  with 
rate  A  and  the  service  time  is  exponentially  distributed  with  expected 
service  time  1/p^  whenever  rate  pfc  is  in  effect.  The  results  of  Section  2, 
Chapter  III  for  the  existence  of  optimal  stationary  policy  are  true  for  K 
rates  case  and  will  not  be  repeated  here.  The  modifications  of  optimality 
conditions  which  were  made  in  the  two  rates  case  will  still  hold  and  will  be 

stated  below. 

* 

If  it  is  the  policy  which  prescribes  the  minimizing  actions  in  the 
following  relations,  then  this  policy  is  stationary  and  minimizes  the  long 
run  expected  average  cost. 

(g  -  A[h(l)  -  h (0)  ] 

g  -  min  {c (p  )  -  p  A  +  A[h(i  +  1)  -  h(i)J  -  p Jh(i)  -  h(i  -  1)]} 

Lf  «  a  A 

i  -  1,2,  ... ,  N-l 

g  -  min  {c(p  )  -  p  A  -  p  [h(N)  -  h(N  -  1)]} 

a/  fl  A  fl 

aeK 

In  the  next  section  we  will  first  eliminate  some  of  the  service  rate  from 
further  considerations  because  they  will  not  appear  in  the  optimal  policy,  and 
then  the  form  of  optimal  policy  will  be  derived. 
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4.2  Elimination  of  Nonoptima]  Rates  and  Derivations 
of  Optimal  Policy 

In  Chapter  III,  Lemma  2,  we  showed  In  certain  cases  some  service  rates 

will  not  be  used  in  the  optimal  policy.  For  example  if  the  function  c(:) 

c(u  ) 

is  concave  and  A  >  . —  for  some  a  ,  then  the  optimal  policy  is  to  use 

^a 

rate  y  ,  the  fastest  rate,  in  all  states.  Lemma  2  is  in  fact  more  general 

lx 

than  this.  If  the  service  rates  are  such  that  the  points  (y^.cCy^))  all 
lie  above  the  line  connecting  origin  and  point  (yK,c(uK))  then,  only  y^ 
will  be  used  in  the  optimal  policy.  We  now  in  a  sequence  of  five  lemmas 
eliminate  some  nonoptimal  rates. 

Let  C  be  the  set  of  all  points  (y  ,c(y^))  ,  and  consider  the  piece- 
wise  linear  convex  function  H  joining  (y^,c(y^))  and  (y^,c(y^))  which 
bounds  the  set  C  from  below  and  changes  slope  only  at  the  points  of  the 
set  (see  Figure  1). 


FIGURE  1:  SET  C  OF  POINTS  (y^.c^))  AND  FUNCTION  H 
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*■ 


Lemma  1: 

The  set  of  service  rates  corresponding  to  the  points  (pk,c(pk))  not 
on  the  function  H  (above  H)  will  not  be  used  in  the  optimal  policy  and 
can  be  eliminated. 

Proof : 

* 

f  (i)  «  k  ,  i.e. ,  rate  yk  is  optimal  in  state  i  if, 
c(yk)  -  pkA  +  X[h(i  +  1)  -  h(i) ]  -  pk[h(i)  -  h(i  -  1)]  < 

(4.2) 

c(pj)  -  p  j  A  +  X[h(i  +  1)  -  h(i)  ]  -  p  [h(i)  -  h(i  -  1))  for  all  j  +  k 


or 


e(pk)  -  P^A  -  c(y^)  J-  vij A  _<  (uk  -  pj)(h(i)  -  h(i  -  1)J  for  J  i  k 


or 


h(i)  -  h(i  -  1)  > 


c(p.  )  -  c(p.) 


(4.3) 


— - -  A  if  uu  >  », 

"k  ‘  "j  k  J 


I  c(p.)  -  c(p.  ) 

h(i)  -  h(i  -  1)  < - i 


-  A  if  p.  >  p. 

-  "j  -  “k  i  k 


(4.3>  can  be  stated  as  follows 


(4.4) 


max 

0-1,2 . k-1 


|c(pk)  -  c(p^) 

I  -k 


<  h(i)  -  h(i  -  1)  < 


min 

J-k+1,. 


c(pi>  -  c(pk) 


Now  assume  that  (uk,c(uk))  is  above  H  ,  and  let  p^  be  the  largest  rate 

to  the  left  of  jj.  .here  (u  ,c(p  ))  is  on  H  and  p  be  the  smallest  rate 
k  m  m  n 

to  the  right  of  p.  an.1  (j  ,c(p  ))  on  H  (see  Figure  2). 

k  n  n 


FIGURE  2 


Then  fro’",  definition  of  H  , 


c(u.)  -  c(u  )  c(u  )  -  c(w.) 
k _ m  _ n _ 

gk  -  %  ”  un  wk 

or 

c(w.  )  -  c(y  )  c(u  )  “  c(p.) 

it  in  n  it  * 

Vk  -  Wo  wn  Mk 

But  this  violates  (A. A).  Therefore,  f*(i)  ■  k  is  not  possible  and 
not  be  optimal  in  any  state  and  can  be  eliminated.  Now  let 


<  ^2  <  • • •  <  those  rates  such  that  the  corresponding  (p,c(p)) 

points  are  on  H  .  (Rearrange  the  indexes  so  that  this  is  true.)  Then, 


We  define 


(A. 6) 


c(u.)  -  c(y  .) 

R,  -  - J -  A 

1  “j  '  “j-i 


Lemma  2 : 


f  (i)  -  j  if  and  only  if 


c(p.)  -  c(y  .)  c(u  .)  -  c(u.) 

(4.7)  - - -  ‘L-  -  A  <  h(i)  -  h(i  -  1)  <  - ^ - J-  -  A 


Pj  "  WJ-1 


Vl  ‘  WJ 


or 


Rj  <  h(i)  -  h(i  -  1)  <  RJ+1  . 

The  proof  of  this  lemma  is  immediate  from  (4.',  d  (4.5). 

The  following  two  lemmas  are  needed  to  prove  Lemma  5  and  Theorem  1. 


Lemma  3; 

Let  (r. ,  ...,  r„)  be  the  optimal  rates  in  states  (1,2,  ...,  N)  for 
1  N 

a  given  cost  function  c(u^)  <_  . . .  <  c(pl)  ,  and  assume  rate  v»t  is  not  used 
In  the  optimal  policy.  Now  let  us  consider  a  new  problem  with  the  same 
structure  but  a  different  cost  function  c'(«)  ,  where 

Jc'O^)  -  cfoj)  i  t  t 
(c’(wt)  >  c(ut)  . 

Then  is  not  used  in  the  optimal  policy  for  the  new  problem. 

Proof : 

The  lemma  is  intuitively  clear  since  increasing  the  cost  of  a  nonoptlmal 
rate  should  not  change  the  optimal  policy.  Formally,  let  6^  be  the  stationary 
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probability,  the  proportion  of  time  the  system  spends  in  state  i  ,  as 
defined  in  Chapter  III,  then 

N 

g  *=  l  B .  c(r  .  )  -  X  (1  -  B„)A 

i«l 

where  g  is  the  minimal  average  cost  rate,  and  (r^.rj,  •••,  r^)  are  the 
set  of  optimal  service  rates.  Now  assume  that  a  different  set  of  rates 
^rI’r2’  rN^  ar°  °Pt*ma^  ror  t^ie  new  problem  and  B^'s  are  the 

corresponding  stationary  probabilities.  Then  if  is  used  in  the  optimal 

policy  in  state  j  ,  g'  the  optimal  cost  rate  for  the  new  problem  is 

N 

g'  -  I  6{c(r!)  -  X(1  -  BA) A  <  g  . 
i-1 

(g  is  an  upper  bound  since  (r-.r.,  ...,  r„)  can  be  used  for  the  new 
problem  which  gives  the  cost  rate  equal  to  g  .)  Then 


g’  -  l  8!c(r!)  +  B!c'(u  )  -  X(1  -  B')A 
iyj  i  i  J  t  N 

(4.8) 

>  l  B!c(r!)  +  B!c(p  )  -  X(1  -  B')A  . 
lflj  l  l  J  t  N 


But  the  right  side  of  (4.8)  is  the  cost  rate  using  the  original  cost  function 
and  gives  lower  cost  rate  than  g  ,  which  contradicts  the  fact  that  g  was 
the  optimal  cost  rate. 

Q.E.D. 


Lemma  4; 

c(Wk)  (c(wa)j 

Let  c  -  — -  -  min  {— - >  ,  a  -  1,2,  . . . ,  L  and  c  <  A  .  Then 

uk  a  (  “a  ) 


(4.9) 


g  >  X (c  -  A)  . 
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*• 


Proof : 

This  is  also  intuitively  clear  since  the  right  side  of  (4.9)  is  the 
negative  of  maximum  return  when  the  cheapest  rate  is  used  and  all  arriving 
customers  are  served.  Let  be  as  defined  in  Lemma  3,  then  if 

(r^.r^  ...,  rN)  is  the  set  of  optimal  rates 

N 

8  ■  l  8tc(r  )  " 

i-1 

N 

1  c  l  re.  -  A(1 
i-1 

The  last  inequality  follows  from  the  fact  that 


N  c(r.) 

VA '  'A  it  * X<1 '  V* 

i-1  i 


6n)A  -  A(1  -  6N)(c  -  A)  >  A(c  -  A) 


Lemma  5 : 


Let 

all  rates 


e(vk) 

:  ■  -  -  min 

Wk  a 

Pj  such  that 


,  a  -  1 ,2 ,  . . . ,  L  ,  and  c  <_  A  then 
can  be  eliminated  from  further  consideration. 


Proof: 


This  lemma  is  a  generalization  of  Lemma  2,  Chapter  III. 

Let  us  assume  for  simplicity  that  k  -  2  .  We  must  show  will  not 
be  used  in  the  optimal  policy. 

Consider  the  same  system  with  the  same  service  rates  available  but  a 
different  cost  function  as  follows: 


c(w2) 

lc*  (w^)  ■  C*^  where  c  -  — -  ,  cCu^)  >  c^ 


c,(ui)  -  c(wi) 


i  -  2,  . . . ,  L 


(see  Figure  3) 


FIGURE  3 

Function  H  :  ABCX  ,  the  cost  function  for 
the  original  problem,  c(*)  . 

Function  H' :  OBCX  ,  the  cost  function  for 
the  new  problem,  c'(»)  . 

We  first  show  that  y^  will  not  be  used  in  the  optimal  policy  for  the 
new  problem.  The  proof  of  the  lemma  then  follows  from  Lemma  3.  In  order 
for  y^  to  be  optimal  in  a  state  i  in  the  new  problem  we  must  have 


c(y.)  -  cy. 

h(i)  -  h(i  -  1)  <  - - - -  -  A  -  c 

-  Wo  -  Ml 


A  . 


By  Lemma  4 


(4.10) 


g  -  X[h(l)  -  h(0)]  >  X (c  -  A) 


A 

:  lien 

h(l)  -  h(0)  *  *  >  c  -  A  . 

* 

Therefore,  f  (1)  j*  1  .  Now  consider  two  cases: 

( ase  1 : 

f  (1)  *  2  ,  then 

g  -  c(y2)  -  m2A  +  A[h(2)  -  h ( 1) ]  -  u 2 [ h ( 1 )  -  h(0) )  . 
by  (4.10) 

g  <  c(n2)  “  W2a  +  A (h(2)  -  h ( 1 ) J  -  m2(c  -  A) 

HU  J 

A(c  -  A)  <  c(n2)  -  n2A  +  A [h (2)  -  h(l) ]  -  p2(c  -  A) 

o. 

A [ h ( 2)  -  h(l)J  >  \ (c  -  A)  +  u 2 (c  -  A)  -  cm2  +  y2A 
h(2)  -  h(l)  >  c  -  A  . 

* 

Therefore,  f  (2)  _>  2  . 

* 

Now,  let  f  (2)  -  t  where  t  >_  2  .  For  state  2  we  have 

g  -  c(ut)  -  ytA  +  A(h<3)  -  h ( 2) J  -  wt[h(2)  -  h(l)] 

<  c(ut)  -  utA  +  A [h (3)  -  h(2)  J  -  . 

Then  by  (4.10) 

A(c  -  A)  <  c(ut)  -  ntA  +  A[h(3)  -  h(2) ]  -  wtRt 


1 


<4.11)  X[h(3)  -  h(2) J  >  X(c  -  A)  +  -  c(pt>  +  v(A 


,tRt  -  c(„t)  +  vt*  -  ;t-Vt;1lc(ut)  -  -  c(ut> 


~t  '  Rt'.1ll‘t-lC<l‘t)  "  VVl’1 


since  for  t  *  2 


wl‘C,Vi2  "  u2*c‘yl  “  0 


and  for  t  >  2 


c(vit)  c(ut_j) 


(H'  being  convex  and  passing  through  origin,  see  Figure  3.)  Hence,  (4.11) 


reduces  to 


h (3)  -  h(2)  >  c  -  A  or  f  (3)  >  2  . 


Continuing  in  the  same  manner  we  can  show 


f  (i)  >  2  for  i  -  3,4 . N 


Case  2: 


f  (1)  ■  t  where  t  >  2  .  Then  h(l)  -  h(0)  >  Rt  >  c  -  A 


X (h (1)  -  h(0)J  >  XR 


»• 


Su 


i’cr  .state  1 


g  *=  c(Mt)  -  wtA  4  A[l.(2)  -  h  ( 1 )  J  -  w ^  lli(l)  -  h (0) ) 


and 


g  <  c(ut)  -  PtA  4-  A (h(2)  -  h(l)  ]  -  utRfc  . 


then 


ARt  <  c <vi t)  -  utA  4  A  (1> (2)  -  h(l)  ]  -  w^Rt 


A (h(2)  -  h(l))  >  ARt  4-  utRt  -  c(tit)  4-  ^A  . 


■  ut 


PtRt  “  +  MtA  -  °  for  t  >  2 

is  was  shown  in  Case  1.  Hence, 

h(2)  -  h(l)  >  Rt  or  f*(2)  >_  t  . 

Continuing  in  the  same  way  it  can  be  shown  f*(i)  >_  t  for  i  -  3,4,  ...,  N 
fills  completes  the  proof  of  the  lemma  for  the  new  problem,  but  by  Lemma  4 
the  same  will  be  true  for  the  original  problem. 

Q.E.D 

By  this  lemma  we  can  now  assume  that  H  passes  through  or  igin  and 


cCmj)  c(m2) 


c<»l> 


i 


< 


< 


•  •  • 


< 
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Theorem  1: 


Let  y,  <  y0  <  . . .  <  y. 


be 


Rj  <  <  .  •  •  <  R^  ,  where  Rj's 

a  nondecreasing  function  of  i  , 


L  service  rates  such  that 
are  as  defined  in  (4.6).  Then 
i.e. , 


f*(i) 


is 


f*(l)  <  f*(2)  1  ...  1  f*(N) 
which  is  a  generalization  of  switchover  policy. 


Proof : 

We  show  that  if  f*(i)  »  t  then  f  (j)  >_  t  for  all  j  >  i  .  We 
consider  two  casr*s: 


Case  1: 


If  f  (1)  =  t  then  by  Lemma  5,  Cases  1  and  2, 


f  (1)  >  t 


for  all  1  >  1 


Case  2: 

*  * 

Now  let  i  be  the  smallest  state  such  that  f  (i)  >  f  (1)  ,  8  >  i  >  1  . 

*  * 

Assume  that  f  (i)  ■  k  and  f  (i  -  1)  ■  t  where  t  <  k  .  For  state  i  -  1 
and  1  we  have 


c(ut) 


c(Vi> 


u-v 


-  A  <  h(i  -  1)  -  h(i  -  2)  < 


c(yt+1)  -  c(yt) 


t-1 


"  Wt+1  •  Mt 


-  A  -  R 


t+1 


c(yk) 


c(pk-l) 


-  y 


-  A  <  h(i)  -  h(i  -  1)  < 


c(Vi>  -  'V 


k-1 


"  Vl  ’  Wk 


-  A 


\+l 


g  -  c(yt)  -  yfcA  +  Xlh(i)  -  h(i  -  1)]  -  yt[h(i  -  1)  -  h(i  -  2) J 
g  -  c(yk)  -  ykA  +  X [h(i  +  1)  -  h(i)J  -  yk[h(i)  -  h(i  -  1) )  . 


But 
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\  i  Rt+i  *  Rt  k  i 1 +i 

then 

(g  ’  c<ut>  -  utA  +  XRk  -  "A 

rg  <  c(wk)  -  yfcA  +  X[h(i  +  1)  -  h(i) ]  -  y^ 

or 


c(wk)  -  yfcA  +  X [h(i  +  1)  -  h(i) ]  -  y^  >  c(yt>  -  yfcA  +  XI^  -  y^ 

(A. 12)  X[h(i  +  1)  -  h(i)J  >  -c(yk)  +  c(yt)  +  ykA  -  yfcA  +  XI^  +  (yk  -  y^  . 


But 


(wk  -  Mt)Rk  -  [e(yk)  -  c(yt)J  +  (yfc  -  yf)A 


C(pk)  -  c 
vk  "  wk-l 


c(yfc)  -  c(yt) 
\  '  yt 


>  0 


the  last  Inequality  holds  by  Lemmas  1  and  5.  Then,  (A. 12)  reduces  to 

h(i  +  1)  -  h(i)  >  ^ 
or 

f*(i  +1)  >  k  .  , 

Continuing  in  the  same  manner  we  can  show  f*(j)  >_  k  for  j  ■  i  +  1, 
i  +  2,  N  ,  which  completes  the  proof  of  Theorem  1. 

c(y  ) 

Now  consider  that  A  < -  for  all  a  ,  then  Lemma  3  will  still  hold, 

y« 

and  the  equivalent  of  Lemma  A  is  Lemma  6. 


Lemma  6: 


Let  c  ■ 


'<V 


g  <  X(c  -  A)  . 


‘  "i"  • 


•  -  1.2, 


L  and  c  >  A  then 
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Proof: 


Let  8^  be  the  stationary  probability  corresponding  to  using  in 


all  states  then 


ai 


1  +  —  +  ...  + 


(t) 


for  i  <  N 


0,-0 


for  i  >  N 


and  let  G  be  the  corresponding  average  cost  rate  then 


g  <  G  -  X(1  -  Bn)(c  -  A)  <  X (c  -  A) 


since  >  0  . 


The  following  lemma  is  the  equivalent  of  Lemma  5  for  the  case 
c(u  ) 

A  < -  for  all  a  . 

u 


Q.E.D. 


Lemma  7: 


Let  c  • 


¥ '  *1"  1 4r( 


and  c  >  A  then  all  rates  faster  than 


>  vk)  can  be  from  further  consideration. 


Proof: 


This  lemma  is  intuitively  clear  since  all  rates  faster  than  are 

more  expensive  and  their  use  will  increase  the  rate  of  number  of  customers 
served. 


The  proof  closely  follows  the  method  to  prove  Leona  5.  First  we  consider 
a  new  problem  with  the  cost  of  rates  faster  than  p.  reduced  to  c*p.  and 

*  J 
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show  for  this  problem  the  optimal  policy  does  not  use  rates  * 

and  then  by  Lemma  3  complete  the  proof.  Details  are  omitted  here. 


Theorem  2: 

Let  <  ^2  <  ”•  <  WL  k®  ^  service  rates  such  that  all  nonoptimal 
rates  are  eliminated  and  R2  <  ^3  <  •  •  •  < 
function  of  i  ,  i.e. , 

f*(l)  21  f *(2)  >_  ...  >  f*(N)  . 

Proof : 

It  will  be  shown  that  if  f  (i)  -  t  ,  then  f*(j)  <_  t  for  all  j  >  i  . 
Details  are  omitted  here  since  method  of  proof  is  closely  related  to  the  proof 
of  Theorem  1  of  this  chapter  and  Theorem  2  of  the  last  chapter. 


then  f  (!)  is  nonincreasing 


55 


REFERENCES 


[1]  Blackwell,  D.  ,  "Discrete  Dynamic  Programming,"  Annals  of  Mathematical 

Statistics ,  Vol.  33,  pp.  719-726,  (1962). 

[2]  Crabill,  T.  B.  ,  "Optimal  Control  of  a  Queue  with  Variable  Service 

Rates,"  Technical  Report  No.  75,  Department  of  Operations  Research, 
Cornell  University,  Ithcca,  New  York,  (June  1969). 

[3]  Heyman,  D.  P.,  "Optimal  Operating  Policies  for  Stochastic  Service 

Systems,"  ORC  66-31,  Operations  Research  Center,  University  of 
California,  Berkeley,  (October  1966). 

[4]  Howard,  R.  A.,  "Research  in  Semi-Markovian  Decision  Structures," 

Journal  of  the  Operations  Research  Society  of  Japan,  Vol.  6, 

No.  4,  (August  1964). 

[5]  Howard,  R.  A.,  DYNAMIC  PROGRAMMING  AND  MARKOV  PROCESSES,  M.I.T.  Press, 

(1960) . 

[6]  Jewell,  W.  S.,  "Markov  Renewal  Programming  I  and  II,"  Operations 

Research,  Vol.  11,  pp.  938-971,  (1963). 

[7]  Klimov,  G.  P.  ,  "Extremal  Problems  in  Queueing  Theory,"  CYBERNETICS  IN 

THE  SERVICE  OF  COMMUNISM,  Vol.  2,  Moscow,  (1964),  Translated  by 
P.  Reed,  ORC  67-17,  Operations  Research  Center,  University  of 
California,  Berkeley,  (May  1967). 

[8]  Lippman,  .  A.  and  S.  M.  Ross,  "The  Streetwalker's  De lemma:  A  Job 

Sh^-p  Model,"  ORC  69-39,  Operations  Research  Center,  University 
of  California,  Berkeley,  (December  1969). 

[9]  McGill,  J.  T. ,  "Optimal  Control  of  Queueing  Systems  with  Variable 

Nunber  of  Exponential  Servers,"  Technical  Report  No.  123, 

Department  of  Operations  Research,  Stanford  University,  Stanford, 
California,  (1969). 

[10]  Miller,  B.  L. ,  "A  Queueing  Reward  System  with  Several  Customer  Classes," 

Management  Science,  Vol.  16,  No.  3,  (November  1969). 

[11]  Mitchell,  W. ,  "Optimal  Service  Rate  Selection  in  Piecewise  Linear 

Markovian  Service  Systems,"  Ph.D.  Thesis,  Operations  Research 
Center,  University  of  California,  Berkeley. 

[12]  Ross,  S.  M. ,  "Average  Cost  Semi-Markov  Decision  Processes,"  ORC  69-27, 

Operations  Research  Center,  University  of  California,  Berkeley, 
(September  1969). 

[13]  Ross,  S.  M.,  APPLIED  PROBABILITY  MODELS  WITH  OPTIMIZATION  APPLICATIONS, 

Holden  Day  Publishing  Co.,  San  Francisco,  California,  (April  1970). 

[14]  Ross,  S.  M. ,  "Arbitrary  Stats  Markovian  Decision  Processes,"  The  Annals 

of  Mathematical  Statistics.  Vol.  39,  no.  6,  pp.  2118-2122,  (1968). 


56 


[15]  Smith,  W.  L. ,  "Regenerative  Stochastic  Processes,"  Proceeding  of  the 

Royal  Statistical  Society,  Series  A,  Vol.  232,  pp.  6-31,  (1955). 

[16]  Stidham,  S.,  Jr.,  "Static  Decision  Models  for  Queueing  System  with 

Non-Linear  Waiting  Costs,"  Technical  Report  No.  1,  Department  of 
Operations  Research,  Stanford  University,  Stanford,  California, 
(1968) . 

[17]  Thatcher,  R.  M.  ,  "Optimal  Single  Channel  Service  Policies  for  Stochastic 

Arrivals,"  ORC  68-16,  Operations  Research  Center,  University  of 
California,  Berkeley,  (June  1968). 

[18]  Yadin,  M.  and  P.  Naor,  "On  Queueing  Systems  with  Variable  Service 

Capacities,"  Naval  Research  Logistics  Quarterly,  Vol.  14,  No.  1, 
(March  1967). 


Unclassified 


DOCUMENT  CONTROL  DATA  -RAD 

iSrrufiiK  rla\ii/irs(lon  of  tttlv.  body  of  sb'TMfi  Anti  u.dexm# »  .uinolHtinn  must  be  fiilfffd  nfirn  the  t.vctnU  report  f%  c  fas  tided) 

i  originating  act  vitv  (Corporate  author) 

M.  REPOET  IECURITV  CLASSIFICATION 

Unclassified 

University  of  California,  Berkeley 

26.  CROUP 

•  HtMOHl  TITLE 


OPTIMAL  DECISION  IN  QUEUEING 


•  LLSCSIPTIVE  NOTES  (Type  ot  report  end,  tnc  lira  ivt  delta) 

Research  Report 


*  au  Thor  (Si  (ht»t  n«m«,  middle  initial,  laat  name) 


Houshang  Sabetl 


April  1970 


BA.  CONTRACT  OR  GRANT  NO. 

DA-31-124-ARO-D-331 

b.  PROJEC  T  NO. 


la.  TOTAL  NO.  OT  PACES  lb.  NO.  OF  REFS 

56  18 


•«.  ORIGINATOR'S  REPORT  NUMICR(I) 

ORC  70-12 


9b.  OTHER  REPORT  NO(SI  (Any  other  number  a  tbmt  may  bm 
this  fport) 


10  DISTRIBUTION  STATEMENT 

This  document  has  been  approved  for  public  release  and  sale;  its  distribution  is 
unlimited. 


M  SUPPLEMENT  ARY  NOTES  AlSO  SUppOTted  by  the  la  SPONSORING  MILITARY  ACTIVITY 

Office  of  Naval  Research  under  Contract  U.S.  Army  Research  Off lce-Durham 

N00014-69-A-0200-1010.  Box  CM,  Duke  Station 

Durham,  North  Carolina  27706 


IJ  AB51RACT 

SEE  ABSTRACT. 


DD  ."?.,1473  (PAGE  I) 

i/H  0l0t-*07-*i11 


Unclassified 

Imrilf  CltiiiTiciiiM 


‘••IS* 


I 


Unclassified 

Security  Classification 


KEY  WORDS 


Variable  Service  Queueing  Model 
Semi-Markov  Decision  Processes 
Discounted  Cost  Model 
Switchover  Policy 


D  .'JT.,1473  (BACK) 

tlll'IIMUt 


